Consistency among Domain Analysts in Selecting Domain Documents and Creating Vocabularies

A study is reported on the consistency of the domain vocabularies created and the source documents selected by domain analysts for domain analysis using DARE (Domain Analysis and Reuse Environment). Consistency was analyzed by measuring the pairwise overlap scores between the domain analysts. The overlap scores of the vocabularies and the source documents were both found to be significantly greater than zero. The effect sizes were large. A positive correlation was also observed between overlap scores of the vocabularies and overlap scores of the source documents. The variability of domain vocabularies created automatically was compared to the variability of domain vocabularies produced manually by domain engineers. The variability of automatic and manual vocabularies was found to be significantly different. The difference was of medium effect size.